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ABSTRACT 

This report evaluates a program which trained 
participants in the instructional design process and indicates how 
effective it was for training instructional design personnel. The 
program was administered to three separate groups on separate 
occasions (N=8^ N=6r N=3) • The program, being self-instructional, 
allowed students to work at their own pace, but they did participate 
in group discussions. The following data were gathered: time to 
complete reading and exercise assignments; errors on workbook 
exercises (constituting criterion-like tasks) ; and student comments. 
Analysis of this data revealed that the program worked well. Student 
comments were generally favorable. The only revisions in the program 
were made on the instructions to students on how to use the program. 
Recommendations are made for further evaluation of the revised 
A program* The appendixes include information on workbook error rates; 
ir^ results of final exercises; and participants' comments on overall 
evaluetion, other models, the handbook, and diagramming. Twelve 
tables of data are presented, (Author/BRB) 
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1.1 



Purpose 



This volume provides a report on the tryout and revision of a program 
designed to provide training In the Instructional development process 



Why the lag? A number of Interrelated reasons suggest themselves. 
The training program, which was evaluated during this period,, runs approximately 
fifteen hundred pages long and requires at a minimum approximately fifty 
hours to complete. For purposes of Instructional effectiveness the program 
had to be administered In such a way that the fifty hours of reading and 
practice could be distributed rather than massed. Because of these time 
requirements. It was difficult to find groups of subjects willing to take 
and complete the whole program. It was only because of the fortuitous 
appointment of the principal Investigator as a visiting professor In the 
Department of Educational Research at Florida State University that the 
tryout and revision process could actually begin and be successfully completedo 

The revision of the program, fairly substantial In scope, also required 
additional time. All major volmunes In the program, USER'S MANUAL, ORIENTATION, 
HANDBOOK, WORKBOOK, and FINAL EXERC I SES underwent revisions. Changes in the 
Instructional development mode minor in scope, were made in just a few 
sections of the HANDBOOK and In WORKBOOK exercises associatfid with them* 
On the other hand, changes in the training vehicle were more substantial in 
scope: There were two types of changes. The more extensive of the two 
Involved the complete revision of the ORIENTATION volume and partial 
revision of the USER'S MANUAL to provide trainees with better 
guidance in the use of the total training vehicle. Other considerably less 
extensive changes. In this Instance of an Instruct I onal nature, were made 
In the HANDBOOK, WORKBOOK, and FINAL EXERCISES. Their purpose was to 
provide Improved guidance for solving the practice problems assigned to 
trainees. Thus, both tryout and revision consumed substantial time as 
did the prior search for a sample of the target audience for which the 



(Cropper, 19/Ia). It supplements a previous report describing the prepara- 
tion of the training program and the Instructional development model It 
contains (Cropper, 1971b),'^ The prior report appeared almost a year and a 
half ago. 
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O . program was designed. 



1.3 



The purpose of this report Is to describe both the tryout and revision 
of the program which was conducted durfng this period. 



What This Report Contains 



The major sections of this report provide the following types of 
Information: 

~ Methods Used to Evaluate the Program 
•« descriptions of tryout sample 
data types 

administrative arrangements 

~ Results Obtained 
raw results 
Interpretation 

— ■ Revisions Hade In the Program 
.« basis for revisions 
• • revisions made 

— • Conclusions 

overall evaluation 
• . recommendations 

The explicit provision of this range of information will make It 
possible for potential users of the program to do their own assessment of 
the effectiveness of the program and of Its usefulness for their own 
purposes^ 
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2. METHOD 



Description of Tryout Sample 

Data Sources 

Administrative Arrangements 



2.1 



Description of Tryout Sample 



K The Target Audience 

The program was designed with a broad target audience In mind varying 
from trainees with no development experience to developers with considerable 
experience. The Intent v/as to recommend to the latter group consideration 
of only the development model (contained In the HANDBOOK). On the other 
hand, the model and the training vehicle were Intended for trainees with 
no experience varying through trainees with Intermediate amounts. of prior 
experience either In classroom or job settings. Selection of a tryout 
sample was based on the Identification of this latter group as the target 
audience. 

2. How the Sample was Obtained 

Three separate groups of students participated in the tryout of the 
program. 

Group I consisted of eight Latin American students receiving Instruc- 
:tlon in development under an A.I.D. program conducted in the Center for 
Educational Technology at Florida State University. Personnel running the 
A.I.D. program volunteered the group as participants In the tryout. The 
Individual students, however, could not be considered volunteers. . This 
phase of the tryout occurred before the principal Investigator assumed a 
visiting professorship at Florida State and before he was offered It. 

Group 11 consisted of six students enrolled In Ph.D. programs at the 
Department of Educational Research at Florida State. They took the program 
as part of a course offered by the principal Invent igator during the Fall 
quarter. The course Is a required course (which Is offered throughout the 
year) In the "Instructional systems" subsection of the Department of 
Educational Research. However, taking the program was one of two options 
offered students who had enrolled for the course. Of the two options, 
taking the program was by far the more time consuming. Three students 
dropped the course. The six who remained and took the program can be called 
volunteers. 



Group III consisted of three students enrolled in Ph.D. programs In 
the College of Education at Florida State University. They took the program 
as part of a course offered by the principal Investigator during the Spring 
quarter. This course dealt with contingency management. The three students 
In Group 111, for varying reasons^ elected not to study contingency manage- 
ment. The principal Invest:! gator offered to run a separate section on 
Instructional technology. The three students In Group III volunteered for 
this option. 

3« Characteristics of the Three Groups 

The general J zabi 1 ity of tryout results based on a sample drawn from a 
target population depends on the degree to which sample and population share 
common characteristics. Since the target population was neither precisely 
defined nor was the sample randomly selected, all that can be done Is to 
describe characteristics of the sample identified on a rational basis as 
being relevant to recommendations for program use. Prior design training 
or experience^ career goals, and levil of prior academic training appear to 
be relevant characteristics. Table 1 on the next page summarizes for each 
of the three groups subject possession of these various, relevant characteristics. 

An inspection of Table 1 reveals that, except for one or two 
characteristics such as prior teaching experience and undergraduate major, 
the total sample on the whole splits mainly down the middle on possession 
or non-possession of each of the relevant characteristics. Almost half 
has had no prior design courses, Is not enrolled In a Ph.D. ''systems'* 
program or has only a B.A. degree. 

Possession of characteristics which might be expected to facilitate 
performance on this new program cannot automatically be judged to have 
exerted a facilitating effect. Having had prior design courses or prior 
design experience cannot automatically be thought either to have provided 
prerequisite skills for Implementing the current, new model or to have 
provided some skills involved In Its implementation. While it Is true that 
differing models do share common strategies and sometimes common tactics 
designed to achieve them. It Is also probably true that models differ 
considerably. For example, students at Florida State have had experience 
with'one model whych differs considerably from the present 
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one both In the amount of procedural detail It offers and In procedural 
content (Briggs, 1970). One key Instance of such differences Involves the 
taxonomy of behaviors used In the models. Since taxonomies are central to 
models, with many other procedural prescriptions hanging on them^ pro- 
ficiency In one model does not Insure or necessarily faci 1 itate.:prof Ic^rency 
In another. And, Indeed, Interference and more difficulty In laasarrnlng 
the second model (the one being assessed In this study) Is a distinct 
posslbl 1 1 ty. 

About half the students In groups II and III have taken the Briggs 
course on Instructional design. Results of program evaluation., Teveal 
superior performance In some areas by students who have had the Briggs 
course (or other courses) and superior performance In other ^a^reas rby 
students who have not had any prior course. Data on this particular Issue, 
is not available In sufficient detail or can they be made in a suf f Ic:Iently 
differentiated way to draw well founded judgments. Perhaps the saSest 
conclusion to draw Is that the prior experience may not have interfered 
with their learning to Implement the present modeJ*,» Also If It had a 
facilitating affect, that effect Is not 1 Ikely ta have been sizeatelie. The 
model was sufficiently new In procedural detail as to have required a 
new and major learning experience. 

The results of groups II and III were ^trlVen greater ^weight dxan those 
of groups I for the following reasons: 

group I, a Latin speaking group, had some difficulty 
with English vocabulary. For that reason they are 
not fully representative of the target audience, 
group I, for a variety of reasons. Including Internal 
politics and squabbles and differences with personnel 
administering their training as to the appropriate 
content of this ♦'.raining did not complete all of the 
program. 

.. although all groups, to use the boldest of language, 
cheated on the program (I.e. copied answers from 
answer pages designed to provide them with feedback), 
group I appears on the basis of Internal evidence to 
have Indulged In this practice to a greater extent. 
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Data Sources 



The prftncipal diata types obtained for the evaluation of tlie fyogram 
are: . 

student performance on program exerclses--ln the 
WORKBDOK and In the FINAL EXERCISES volume 
•• student written and oral comments on program 
features and their experience In taking the 
program. 

Performance data, primarily errors, were based .on student self-scoring and 
self-reportlTTg^ (for groups ff and III) and based on supervisory scoring and 
reporting (for group I). Oral and written comments were obtained during 
weekly class meeitrngs from groups II and III. Comments served to Identify: 

.. areas In the HANDBOOK, WORKBOOK, or FINAL EXERCISE 

volumes^ where students identified difficulties 
.. administrative difficulties experienced in going 

through the program 
.. program features they found helpful or non-helpful 

(both with respect to model prescriptions and with 

respect to the training vehicle). 

Administrative Arrangements 

! 

1. Standardized Administrative Arrangements \ 

Instructions for taking the program appear in the USER'S MANUAL and In 
the ORIENTATION volume. Additional detailed scheduling instructions also 
appear in the WORKBOOK and the FINAL EXERCISES volume. Taken together, 
instructions, no matter In which volume they appear, are designed to guide 
students or trainees through the following standardized routine: 

.. a subject reads the USER'S MANUAL and the 

ORIENTATION volume; following instructions which 
appear in the latter volume he actively Inspects 
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all program components and Identifies both 
the kinds of information they provide and the 
methods recommended for using them • 
••a subject follows the schedule of events Identified 
In. gross detail In the USER's MANUAL and In specific 
detail on divider pages In the WORKBOOK. He reads 
an assigned portion of the HANDBOOK sub-volume he 
Is working on and then does the WORKBOOK exercises 
associated with It; as he completes each exercise 
he Inspects answer pages provided to give him feed- 
back as to the correctness of his own responses; 
when he has completed all the exercises associated 
with a given reading assignment, he then moves on to 
the next reading assignment and Its associated 
exercises • 

o. when the subject completes al 1 reading assignments 
and all WORKBOOK practice assignments associated 
with a HANDBOOK sub-volume, he then does the FINAL 
EXERCISE associated with that HANDBOOK sub-volume« 
following completion of the FINAL EXERCISE, the 
subject resumes the cycle of reading and practice 
, for the next, regularly scheduled HANDBOOK sub- 
volume. 

this routine continues until all the HANDBOOK 
sub-volumes have been read, all the WORKBOOK 
exercises completed, and all the FINAL EXERCISES 
completed. 

Except in the case where attrition to the sample occurred (to be 
described later), all tryout subjects followed this standard routine. 

Additions to this routine are described below for each of the three 

/ 

participating groups. 




2. Group I Arrangements 

\ 

The eight subjects in group I took the program following the routine 
described above. They worked at their own pace* Although the program was 
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designed to be self-instructtonal , subjects In this group did interact 
with one another and with members of the A.I.D. project staff. Subject 
of their discussions were difficulties, they experienced with procedural 
instructions, vocabulary, or program content. Resolution of these 
problems occurred after subjects attempted to negotiate the program 
themselves. However, resolution of problems occurred frequently enough 
and at enough points in early portions of the program so that problems 
were less likely to be cumulative and less likely to recur in subsequent 
portions of the program. Under such arrangements, the program can be 
said to have been self-Instructional in only a limited sensep How portions 
of the program scheduled late in the sequence would have fared without such 
local f personal guidance and correct Ion remains indeterminate. 

3> Group H and Group lit Arrangements 

Administrative arrangements for group II and III were under the control 
of the principal investigator. In most respects they paralleled the 
arrangements which were used with group I. 

The program was administered in weekly course segments. While subjects 
were allowed to adopt their own pace, they were required to complete 
weekly assignments. The most frequently occurring type of assignment 
involved the reading of an entire HANDBOOK sub-volume and the completion 
of both the WORKBOOK exercises and the FINAL EXERCISE associated with it. 
Thus, subjects could pace themselves within the time available during 
a week's Interval . 

Following the completion of each weekly assignment subjects met with 
the principal investigator for a regularly scheduled three hour discussion 
session. It was at this time that subjects provided oral and written 
comments about the program and about difficulties (or lack of It) they 
experienced. Subjects also critiqued the products of one another's 
FINAL EXERCISE activities. 

The learning experience which groups II and III underwent can be 
described as having been composed of a combination of self-Instruction , 
plus group discussions, plus evaluation by the principal investigator. 
As was the case for group I, the evaluation of program segments late In the 
sequence has to take into account the effect of guidance and correction 
which occurred earlier In the sequence. 
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3.1 



Introduction 



For purposes of answering what kinds of questions were data collected? 
Were the data actually collected suitable for those purposes? This latter 
issue will be the subject of the last, Interpretive section of this chapter 
on results. Here, those major Issues relevant to the evaluation of any 
program Intended to provide training In Instructional design will be Identi- 
fied. They are the Issues of: (I) model adequacy; (2) training vehicle 
adequacy; and (3) administrative arrangements adequacy. 



K Model adequacy 

The issue of model adequacy does not concern the effectiveness of the 
program used to train instructional designers. It does concern the adequacy 
of the program content they are being taught. It concerns such interrelated 
questions as: , 

. . Is the design model, that is to say, the prescriptions 
or procedural rules for designing instruction, a 
valid model? Is it relevant to all the kinds of 
objectives developers will have to teach? 

. • Assuming the adequacy of the vehicle employed to 
train subjects in the implementation of the model, 
can the model be implemented In a consistent or 
reliable way? Do independent subjects produce 
comparable development products? 

. • When consistently implemented, does the implementation 
of the model produce instructional programs which work? 
When subjects follow the procedures required by the 
model, do they produce programs which effectively 
teach their students? 

It will be of interest to determine whether the data collected can 
answer these questions. 
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2. Tratning vehicle adequacy 

Empirical tryout of a program Is designed primarily to answer the 
questions of training vehicle adequacy. It concerns such Interrelated 
questions as: 

. . Are the practice tasks which make up the program 
relevant to the goal of preparing subjects to 
Implement a particular development model? 

. . Does the training vehicle do an effective job of 
training subjects to Implement the model? 

. . If there are Inadequacies in the training vehicle, 
what are they? Which specific portions of the 
vehicle are effective and which Ineffective? Which 
training needs are achieved and which are not? 

. . If portions of a training vehicle are Ineffective, 
what accounts for that Ineffectiveness? What kind 
of revision will increase their effectiveness? 



3« Administrative arrangements adequacy 

A training vehicle cannot demonstrate Its effectiveness unless the 
vehicle Is used properly. Accordingly, there are Important questions to 
ask concerning the adequacy of Instructions about program administration: 

. . Are the recommended procedures for going through 
a program followed correctly and consistently? 

. . If there are procedural failures, what are they 
and to what properties of the instructions 
provided can account for them? 



The next four sections of this, the results chapter will provide the 
raw data collected. The final section of this chapter will Interpret their 
relevance to these questions and will provide those answers they are capable 
of providing. 
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Results of WORKBOOK Exercises 



The entire WORKBOOK contained a total of 589 scorable responses. Some 
of these responses Involved the simple endorsement of multiple choice 
options. Other responses, often much more complex and considerably larger 
In scope, required subjects to produce their answers. Thus, while the number 
of responses may appear to be smalli considering the scope of the competencies 
the program is designed to foster, awareness of the cr I ter lon~l i ke complexity, 
scope, and difficulty of many of the responses leads to a fairer appraisal 
of what the program demanded of subjects taking it. Table 2 indicates how 
the 589 responses were distributed among the various sections of the 
WORKBOOK. Each of the separate sections Is associated with a major task 
in the development process. The relative number of responses per task does 



Table 2 

Distribution of Practice Responses 
Among Major Development Tasks* 



HANDBOOK 
Subvotume 


TASK 


Number of 
Responses 


B 


analysis of criterion behaviors 


77 


C 


sequencing behaviors 


15 


D 


stating objectives 


kk 


E 


planning simulation 


58 


F 


developing tests 


5A 


G 


formulating instructional strategies 


152 


1 developing instructional materials 




J 


trying out and revising materials 


112 



There were no exercises associated with Tasks A and H. 

in part reflect scope. But In view of the comments made earlier. It can be 
fairly said that the distribution also conceals what Is expected of the 
subject. Ten responses in one area may be more demanding than twenty or 
thirty in another. 
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There is littJe point In providing a detailed breakdown of types. 
Suffice It to say that the number and type of responses built into WORKBOOK 
were precisely what the developers of the program judged to be necessary 
to teach instructional design skills. The success of the program in teaching 
those skills (e.g., on FINAL EXERCISES) can attest to the adequacy of that 
judgment. 

Table 3 on the next page provides a summary of the overall error rates 
on WORKBOOK exercises associated with each of the eight of the ten. develop- 
ment tasks for which exercises had been prepared. 

Appendix A provides an error rate breakdown for groups of exercises 
associated with steps or sub-steps all of which comprise each major tasks. 
It Is this latter kind of more detailed information which served as the 
basis for revision of HANDBOOK sections. Data for groups of exercises 
associated with sub-sections of the HANDBOOK were used to evaluate the 
adequacy of the sub-sections to cue correct practice of specific development 
activities. More detailed response by response analyses were also performed 
and served as a basis for revision of specific WORKBOOK practice problems. 
This latter type of detailed results on each practice item is no-t reproduced 
In this volumeo 

Returning to Table 3 which summarizes WORKBOOK error rates associated 
with each of the eight HANDBOOK subvolumes for which there are practice 
exercises, each numerical entry represents the percentage of subjects in 
each of the tryout groups scoring at one of these error-rate levels: 
zero error rate, less than 20% error rate, and more tha 1Q% error rate. 

The data are tabled to answer two primary questions: (1) Is there 
consistency among the three groups In the error rate patterns?, and (2) 
With usual developmental tryout standards in mind, how do each of th'e major 
sections of the program fare? In answering these questions, entries for the 
rows <2Q% (less than twenty percent) and 0% can be combined. This results 
in a combined figure indicating error rates of less than 20%. Settling on 
this cutting point is justifiable on at least two counts. First of all, 
it is not an reasonable standard for purposes of tryout of instructional 
materials during developmental or formative evaluation. Secondly, because 
the size of all three tryout groups is relatively small, a difference of 
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one additional subject, tends to make error rates jump inordinately. Under 
these circumstances, attempting to adhere to the oft cited 10% error rate 
would be next to Impossible. For both reasons, *'Less than 20%" is the 
standard set. 

On this standard, how do the three tryout groups compare? Inspection 
alone reveals remarkable high consistency among the three groups. This 
consistency obtains for all major tasks. Whatever deviations there are are 
more likely to be attributable to the small sample size and the sizeable 
increases or decreases in error rates due to a change in performance of 
one tryout subject. y 

How do the exercises associated with the different tasks compare? 
By and large, the percentage of subjects scoring at less than a 20% error 
rate is quite high. This result holds for most of the eight tasks — with 
TASK E being the major exception*. Table k provides a summary of weighted, 
average percentage (across groups) of subjects performing at l,ess than a 
twenty percent error rate. This is done task by task. 

Table k 

Average Percent of Subjects Performing 
at less than a Twenty Percent Error Rate 



TASK 


Average 
Percentage of Students 


J 


11% 


1 


90% 


G 


82% 


F 


77% 


E 




D 


95% 


C 


88% 


6 


93% 
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Except for TASK E, percentages are relatively high Indicating medium to 
high program effectiveness. In passing^ It should be noted that the practice 
exercises In TASK Eiare highly judgmental In nature. Error rates reflect 
discrepancies between subject's ratings and program set ratings. Since the 
latter are also subjective, it Is not unusual for differences In rating 
levels to occur as between different raters. 

The more detailed result In Appendix A and the still more detailed 
results (for each practice problem) served as the basis for revision of the 
highly 'MocaT' learning difficulties generated by specific sections of the 
program. The pattern of WORKBOOK results summarized In the two tables 
above suggest no generalized or widespread program Inadequacies. 
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Results of FINAL EXERCISES 



With the completion of all the separate WORKBOOK exercises associated 
with a particular TASK, subjects moved on to a FINAL EXERCISE for that 
task. In the final exercise, subjects were expected to perform all the 
major routines and sub-routines, heretofore practiced separately, in their 
entirety and in sequence. Each FINAL EXERCISE generally required subjects 
to perform a specific, total task and the next task in the development 
process. Thus, there was generally more than one opportunity to practice 
each task. For example, although there was only one FINAL EXERCISE associated 
with TASK G,' Formulating Instructional Strategies,' there were actually 
two opportunities to practice this particular, entire task. Results pro- 
vided below will identify the number of times each major task was practiced. 

Each FINAL EXERCISE involved the completion of specific forms used in 
the implementation of a particular task. Scoring of FINAL EXERICSES, on 
the one hand, consisted of an assessment of whether or not a form was 
correctly filled out; that is, were all the blanks filled out. This decision 
on adequacy was done simply on a yes/no basis. It was a decision about 
procedure. Scoring also consisted of assessing the validity of entries. 
Since content could vary and still be valid, adequacy was again judged on 
a simple yes/no basis. 

Tables 5-8 summarize results of this type of analysis only for groups 
II and III. (Subjects in Group I either did not do the final exercises at 
all, or did not complete them all, or it is suspected they did them after 
first checking illustrative 'answers'). Tables 5 and 6 summarize percentages 
of students doing each exercise correctly. Tables 7 and 8 disregards in 
which exercise each type of task was done and summarizes the results for 
each type of task* 
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Table 5 

Percentage of Students In Group 
Correctly Performing Development Tasks on Final Exercises 

n»6 Students 



Type of Act ivl ty 


Performed in 
^ Exercise 


FORM 
Correctly used 


Content 
Val tdrty 


J 


Analysis of tryout results 


#! 


100%* 


100%* 


J 


Revisions of instructional program 


U 1 


N.A. 




1 Development of a program 


#2 




83% 


1 


Development .of a program 


#3 


100% 


100% 


G 


Formulating a strategy 


#3 


100% 


100% ■ 


G 


Formulating a strategy 




100%* 


100%* 


F 


Constructing a test 




100% 


100% 


F 


Constructing a test 


#5 


100% 


100% 


E 


Simulating 


#5 


100% 


100% 


E 


Simulating 




100%** 


100%** 


D 


Stating objectives 


#6 


100% 


83% 


D 


Stating objectives 


a ■ 


100% 


67% 


D 


Stating ob Jectlves 


#8 


100%* 


100%* 


B 


Analysis of behavior 


#7 


100% 


50% 


B 


Analysis of behavior 


#8 


100% 


100% 



* N=5 
** N=3 
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Table 6 



Percentage of Students In Group I IJ 
Correctly Performing Development Tasks on Final Exercises 

n»3 Students 





Type of Activity 


Performed in 


FORM 


Content | 






Exerc ise: 


Correctly Used 


Validity 


J 


Analysis of tryout results 


#1 


N.A. 


1 

100% 1 


J 


Revisions of instructional program 




N.A. 


100% 
A 


1 


Development of a program 


#2 


83^ 


83% 


1 


Development of a program 


#3 


100% 


100% 1 


G 


Formulating a strategy 


#3 


100% 


100% 


G 


Formulating a strategy 




100% 


100% 


F 


Construct ina a test 


ftk 


100% 


100% 


F 


Cons t rucf! i na a test 




100% 


100% 




Simulating 


#5 


100% 


100% 


E 


Simulating 


#6 


100% 


100% 


) 


Stating objectives 


#6 


100% 


83% 


D 


Stating objectives 


#7 


100% 


67% 


) 


Stating objectives 


#8 


100% 


100% 


B 


Analysis of behavior 


#7 


100% 


50% 


B 


Analysis of behrivior 


#8 


. 100% 


100% 
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Table 8 



Percentage of Exercises Devoted to a Development Activity 
Correctly Performed by All Subjects in Group 1 1 I 

SUBJECTS 



TASKS 


Form Content 


Anal vs 1 s of t rvou t resu Its 
n=»l 




1 nn2: 


Program revision 
n«1 


100% 


100% 


Program development 
nal 


100% 


67% 


Formulating strategies 
n=»2 


100% 


100% 


Constructing tests 


100% 


100% 


S i mu 1 a 1 1 on ' 
n=2 


100% 


100% 


Stating objectives 
n-3 


100% 


55% 


Analysis of behavior 
n-2 


100% 


83% 



n=number of practice opportunities 
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Appendix B provides the detailed results which tables 5-8 summarize. 

Inspection of tables 5-8 reveals that, with a few exceptions and then 
only to a minor degree, achievement for all final exercises and for all 
subjects was at a high level* 



Time Data 

The potential user of any program needs to know how much instructional 
time will be required to complete the program. For that purpose, subjects 
in Group II (n=6) were asked to keep records of time it took them to read 
assigned HANDBOOK subsections and to do the WORKBOOK practice exercises 
associated with them. Tables 9 and 10 provide such time data 7n hours 
and table II provides times of the two activities combined. 
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Table 9 

Report Reading Time (In Hours) 
for HANDBOOK Sub-volumes 
(Group II) 

Sub-Volumes 



Sub ject 
# 


J 


1 


H 


G 


F 


E 


D 


C 


B 


A 


1 


5.0 


'♦.o 




2.8 


1.7 


0.5 


0.8 


1.5 


2.5 




2 


6.0 


2.0 


0.8 


'♦.5 


1.2 


0.8 


1,2 




1.8 




3 


2.5 


1.3 


0.3 


2.0 


1.1 


0.5 


1.0 


0.5 


1.9 


0.6 


k 




1.5 


0.5 


2.2 


2.5 


0.8 


1.3 


0.5 


3.5 




5 


1.9 


0.7 




2.0 


1.1 


0.8 


1.3 


0.6 


1.8 





Al 1 
Volumes 



18.8 



18.3 



11.7 



17.8 



10.2 



Mean 
Time 


3.8 


1.9 


0.5 


2.7 


1.5 


0.7 


l.l 


0.8 


2.3 


0.6 


Range 


Low 


1.9 


0.7 


0.3 


2.0 


1.1 


0.5 


0.8 


0.5 


1.8 




High 


6.0 




0.8 


'♦.5 


2.5 


0.8 


1.3 


1.5 


3.5 





o 
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Table 10 

Reported Work Time (in Hours) 
for WORKBOOK Exercises 
Associated with each HANDBOOK Sub-Volume 
(Group I I ) 

Sub-Volume 



Subject 

# 


J 


1 


H* 


G 


F 


E 


D 


C 


B 


A* 


1 


3.0 


2.8 




3.6 


1.0 


0.7 


0.8 


0.5 






2 


1.8 


1.3 




3.6 


0.7 


1.0 


0.8 




1.2 




3 


1.1 


1.0 




2.1 


0.8 


0.6 


0„6 


0.5 


1.6 




k 


1.5 


0.8 






1.3 


0.5 


0.8 


0.8 


1.5 




5 


1.5 


1.1 




1.0 


1.0 


0.1* 


0.8 


0.3 


1.3 






Mean 
Time 


1.8 


].M 




2.3 


0.9 


0.6 


0,8 


0.5 






RANGE 


Low 


1.1 


0.8 




1.0 


0.7 


OA 


0.6 


Oo3 


1.2 




High 


3.0 


2.8 




3.6 


1.3 


1.0 


0.8 


0.8 


1.6 





* No exercises for these subvolumes 
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Table 11 



Combined Reading and Work Times 
for each Subject 
(fn hours) 





Read 1 ng 


Work 


Total 


#1 


18.8 


12. A 


31.2 


#2 


18.3 


10. A 


28.7 


#3 


11.7 


7.6 


19.3 


»k 


17.8 


8.6 


26.A 


#5 


10.2 


7. A 


17.6 



10.2 


7.A 


17.6 


18.8 


12. A 


31.2 
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From Table 11, ft can be observed that the smallest completion time 
was hours, the largest 31.2 hours. To these totals must be added the 

times required to read the USER'S MANUAL, to do the ORIENTATION exercises, 
and to do the FINAL EXERCISES. There are no systematic data available for 
these latter activities. Based on student comments and fragmentary data. 
It Is estimated that their time requirements are as indicated In Table 12. 



Table 12 

Minimum Estimated Time 
Required to Complete 
Three Program Activities 
(In hours) 



ACTIVITY 


MINIMUM 
ESTIMATED 
TIME REQUIRED 


Reading the 
USER'S MANUAL 


1 


Doing the ORtENTATION 
exercises 


2 


Doing FINAL 
EXERCISES: 
(9 exercises) 
§ 3hrs/exercise 


27 



Adding this additional estimate of thirty hours to the high and low levels 

of obtained time data. It appears that the total time likely to be required 

to complete all program activities ranges from a low of approximately ^7 
hours to a high of 61 hours. 
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Subjects' Comments 



Group II (n«6) met with the prSncipal Investigator on a weekly basis 
to discuss their weekly assignments. Subjects In this group, as part of 
their assignment, provided both written and oral comments on the model, 
on the training vehicle, and on administrative arrangements. Because of 
this constant interaction between tryout subjects and the developer, the 
Group tl tryout, in addition to its developmental tryout functions, was 
also able to capitalize on the properties of tryout only available in infor- 
mal tryout. 

Subjects were able to comment on any and all features of the program, 
identifying among other things: 

typographical errors 
examples which were unfamiliar 
— — missing Information 

— unclear instructions about scope, type of assignments 

or response 

— — omissions or inadequacies In the model (according to 
their judgment) 

difficulties with reading sections In the HANDBOOK 

■ exercise difficulties 

No attempt has been made to quantify the information obtained from, 
students comments. However, all student comments. In their entirety and 
with no omissions, are reproduced in Appendices C, D, E, and F. These 
provide respectively: 

— an overview evaluation of the model and the training 

vehicle 

— a comparison with other models (with which subjects 

are faml 1 iar 

— an evaluation of specific subvolumes (some volumes 

not evaluated) 



an evaluation of the diagramming techniques used 
throughout the HANDBOOK 
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Further references to subjects' comments will be made in the final inter- 
pretive section of this chapter. 

Interpretat ion 

1 • M odel adequacy 

Keeping in mind the disctinction between the content of an instructional 
program (in this instance the instructional design model being taught) and 
the vehicle used to teach the contentj^ it is safe to say that performance 
data (e.g. program errors) can in only a very limited way answer questions 
about the adequacy of the design model. It is for research and/or applied 
efforts, not developmental tryout, to answer questions about: the relevance 
of the model to the range of objectives likely to be taught or the effective- 
ness of instructional programs produced according to model prescriptions. 

It would have been possible to have subjects prepare programs 
(according to model prescriptions) and to assess the instructional adequacy 
of those programs. This was not done here. Even had it been done, such 
global assessment would be insufficiently diagnostic of model strengths 
and weakness. Programs which fail can do so for any number of alternative 
weaknesses in a design model. Moreover, the requirements of tryout and 
revision tends to gloss over program inadequacy. It would appear to be 
appropriate in assessing design models to consider instructional effective- 
ness prior to revision and to consider how much revision is required and 
how many cycles of revision are required. 

Subjects' performance, while not suitable for assessing model validity, 
is suitable for assessing the reliability with which students implement 
the model. The consistently high achievement levels attained by all subjects 
in FINAL EXERCISES, the most job-like tasks to be found in the program, do 
suggest thst the model can be reliably implemented. 

Subjects' attitudes, as expressed through their comments, are not 
suitable data in assessing model relevancy or effectiveness. Empirical 
data are. However, subjects' attitudes are suitable for assessing the like- 
lihood of a model (as opposed to other models) subsequently being implemented. 
A review of subjects' comments to be found in Appendices C, D, and E do 
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suggest highly favorable attitudes toward the usefulness of various portions 
of the model. There were of course Individual differences among subjects 
in their preference for particular parts of the model, A common negative 
attitude expressed concerned the treatment of 'Individual differences.' 
Subjects apparently had more confidence in the currently available evidence 
concerning aptitude— •treatment--!nteract?ons than did the principal investi- 
gator. 

In summary, the most that can be said about the adequacy of the 
development model, based on tryout data which Is not generally suitable for 
assessing models, is that model procedures can be reliably Implemented and 
that tryout subjects have faborable attitudes toward portions of the model 

(there being Individual differences was to which portions Individual subjects 

favor) . 



Training Vehicle Adequacy ^ ^ 




Based both on subjects' performance and comments it is possible to 
make the following conclusions about the adequacy of the training vehicle. 

— - students in Group M had an initial appreciable degree 
of difficulty In making use of the novel diagrammed 
formats which appear In the HANDBOOK; this difficulty 
diminished with increased use. 

— - students in Group Ml (Who had been exposed to a revised 
ORIENTATION volume and who had been required to read 
Volume "A" for orienting, overview purposes 
(also a revised procedure) had less difficulty getting 
started In working with diagrams. 

— The relatively low error rates on both WORKBOOK exercises 
and on FINAL EXERCISES suggest that the amount and type 
of cueing provided in both the HANDBOOK and in the WORK- 
BOOK are adequate; and 

The relatively low error rates on FINAL EXERCISES suggest 

that the cummulatlve practice experience was adequate 

to build criterion proficiency In Implementing the design 

model being taught. 
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Administrative Arrangements 



Based on administrator comments (Group I) and subject comments, It is 
possible to make the following conclusions: 

— There was little difficulty in following the instructions 
for the overall routines prescribed for the use of the 
program; 

Some initial difficulty was experienced in identifying the 
prescribed, specific sequencing, listed in the WORKBOOK, 
to follow in reading HANDBOOK subsections and doing, 
exercises associated with them; this difficulty diminished 
as subjects continued on to later assignments; and 

Some Initial difficul ty was- experienced in performing the 
first FINAL EXERCISE with subjects not knowing what was 
provided as information and what the practice arsignment 
was; this difficulty also diminished as subjects progressed 
to subsequent assignments. 
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k. PROGRAM REVISION 

page 

• • Introduction 4.3 

. . Changes In the Development ^^odel .... 4.3 
. . Changes In the Training Vehicle . . • . k.k 
. . Changes In Administrative Arrangements . 4.8 




I ntroduction 



Changes were made In all the volumes which make up the program. Some 
changes were major, others mlnor^ They were made based on problems Identified 
on the basis of both subjects' performance and comments. In the sections 
that follow revisions which are discussed will be related to problems that 
had been Identified rather than simply in terms of which volume was 
revised. 

Revisions In the Design Model 

Revisions were made In the design model (which is presented in the 
HANDBOOK volumes). These changes were based, not on considerations arising 
out of tryout results (which, as pointed out, were not relevant to the issue 
of model adequacy) but on rational considerations. 

Only one change was made In the model. The change consisted only in a 
change in terminology. It represented a switch from new, idiosyncratic 
terminology to conventionally used terminology. Essential distinctions were 
maintained In the change and only labels for categories being distinguished 
were changed. Specifically, the original model referred to three types of 
objectives: criterion, modified preparatory, and unmodified preparatory 
objectives. The new terminology became: criterion, sub-cri terlon, and 
preparatory objectives. (Labeling changes in HAMDBOOK sections on types of 
tests were made to parallel these changes.) These changes br^ng the terminology 
into line with that found in the literature (with ^'preparatory" objectives 
being the counterpart of "enabling" objectives). There had been an expecta- 
tion that the original terminology would serve "surplus" functions in guiding 
development procedures. However, confusion due to terminology prompted the 
change - with no expected loss In the guidance value attributable to the 
distinctions being made. 
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Revisions In the Training Vehicle 

A number of revisions which have been made are Intended to solve one or 
more Identified problems. Accordingly, they may be referred to several times 
below. 

I . Revisions due to problems created through the use of backward 
chaining 

Tasks In the development process are taught In the reverse order 
from that In which they are usually performed. As a result, a number of 
concepts, procedures, forms, etc., which are Introduced In volumes for 
tasks earlier In the overall design process, are referred to fn the 
later volumes without explanation. To enable the learner to deal with 
them, a number of changes were made. These are listed below. 

a. A glossary 

A glossary was prepared containing key terms which are used 
throughout the several volumes of the HANDBOOK (the basic guide 
to the design model). The glossary, which appears In the beginning 
of the Index subvolume of the HANDBOOK, provides a definition for 
each term and cites sections of the HANDBOOK for additional 
Information on It. 

The glossary was available for use by group III. From their 
comments. It Is apparent that subjects In this group did use It 
for the purposes It was Intended to serve. 

b. A procedural, overview map 

A diagrammed map of all the procedural steps In the design 
process was prepared and also Inserted at the beginning of the 
Index subvolume of the HANDBOOK. Its purpose was to Identify where 
In the overall design process a subject Is, when he Is performing a 
particular task, or a step, or a sub-step. This orienting map 
ts available for use as a learning aid when the subject is working 
backward while learning to design Instrumental materials or as a 
Job aid when the subject Is working forward v^lle doing on-the-job 
design tasks. 
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c- Hore guidance In FINAL EXERCISES 

When performing a FINAL EXERCISE on a particular task, a subject 
has available to him completed forms representing the output of 
tasks which come earlier In the design process. But these are 
tasks he has not learned to perform. Thus, completed examples of 
forms used In these tasks represent unfamiliar materials. To 
offset this problem, additional cues were prepared to call attention 
to particular features of the forms or of the content presented 
in them. Such cues were designed to familiarize subjects with the 
forms themselves and their content and to orient them to their 
relationship to forms involved In prior and In subsequent tasks. 
Forms which, without such cues might have been obscure and meaningless, 
are thus now more likely to be more readily accessible. 

d. A change In the sequence of assignments 

The basic "'backward chaining" sequence calls for subjects to 
work from tasks J, I, H, etc. backward in the design process. In 
the revision this sequence is maintained. However, subjects are 
now Instructed to read HANDBOOK subvolume "A" (for which there are 
no exercises) before starting on "J." This Is largely for orienting 
purposes. All the forms used fn tasks B-J are Introduced In the 
'*A'* volume. Thus, subjects have a chance to familiarize themselves 
with them so that when they encounter them in backward chained FINAL 
EXERCISES they no longer are unfamiliar forms. 

2. Revisions due to problems created by unfami 1 iar i ty with diagrammed 
formats , 

a. The ORIENTATION volume 

The HANDBOOK, which was designed to serve both as a learning 
aid and as a job aid, uses diagrammed formats throughout. For 
subjects accustomed to presentations In connected discourse formats, 
this novel presentation format took some getting used to. 
Particularly, subjects were Initially confused about whether to 
read down columns or to read across rows. They were Initially 
uncertain about how much of the detail they needed to Inspect. 
They were Initially unclear as to how the different types of 

^.5 
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diagrams were to be used. To clear up these various problems, 
the ORIENTATION volume was completely revised. 

Where the original version had subjects only Inspecting 
portions of the HANDBOOK on instruction, the revised version 
precedes this Instruction with guided inspection of Illustrative 
diagrams. Diagrams are reproduced in the ORIENTATION volume 
itself. The text now calls attention to specific features of 
various types of diagrams and to how they are to be used. More 
detailed guidance is provided on what the subject is expected to 
get from each diagram. Subjects are reassured that memorization 
(a frequent subject preoccupation) of detail is not necessary. 
Rather, Jt is the concept(s) or princlple(s) which Is important. 
Once subjects feel self-assured that they have unders tood the 
concept or principle, they are free (the instructions state) to 
read additional material on a given diagram or to go on to another -* 
whichever they choose to do. 

b. The Administrative Manual 

To reinforce the point about "understanding'* rather than 
memorizing. Instructions in the administrative manual also stresses 
the point. 

3. I^evislons due to problems created by insufficient cueing 

The patterns of errors on exercises, bbth In the WORKBOOK and 
FINAL EXERCISES volumes, suggested no serial cumulative difficult 
ties. Whatever difficulties there were, were local and isolated. 
Scattered, minor revisions were made in the HANDBOOK: the addition, 
omission, or revision of a page; the addition of or substitution 
of an example; and the addition of or reinstatement of a rule^ 
principle, or definition. WORKBOOK revision also consisted of 
scattered, minor revisions: task Instructions were changed; an 
example was revised or a new example was substituted; and cues 
were changed or added. 

On the whole, due to relatively low error rates, the percentage 
of material altered is difficult to estimate quantitatively, is 
judged to be low. On the conservative side, It Is possible to 
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estimate that less than ten percent of the HANDBOOK and WORKBOOK 
was revised. 

k. Revisions due to problems created by failure to provide a sufficient 
overview of the design process . 

Whenever the behavior to be taught fs complex and Involves 
many routines and subroutines, it becomes necessary to teach the 
behavior part by part. Eventually, the parts must be brought 
together and interrelated. Based on subjects' comments, it became 
apparent that because of the sheer scope of the model being taught, 
subjects working on isolated parts of the design process were 
having difficulty keeping track of where in the total process 
they were. A number of revisions were made to deal with this 
problem. 

a. A change In sequence 

As already noted, the reading of HANDBOOK sub-volume '*A'* 
was rescheduled. It became the first assignment, now occurring 
prior to practice involving sub-volume **J*\ The inspection of 
all diagrams Involved the various tasks in the design process 
(task analysis, stating objectives, simulation, test construe- 
tlon, etc.), now provides not only for the Identification of 
forms which subjects will eventually use themselves but also 
for an overview of the entire design process. 

b. An overview map of the design process 

The Index sub-volume (of the HANDBOOK) provides an 
overview page for each STEP In the design process. Each such 
page lists all the SUB-STEPS involved In a STEP. Subjects are 
instructed to keep this page open before them as they work on 
a particular STEP. Thus, v^lle working on a SUB-STEP within 
that STEP, they can easily Identify the SUB-STEPS which preceded 
tt and those that will follow it. No change was made in th is 
provision. 

This represents revisions based on performance. As noted earlier, 
some changes were made In the HANDBOOK due to the decision to make 
revisions In the design model > 
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A new, summary map was created and inserted at the 
beginning of the Index sub-volume. It provides a more global 
listing of TASKS and STEPS. It also Identifies the outputs 
of particular tasks and steps which become Inputs to other 
tasks and steps. This ''map" Is designed to provide an 
overview for subjects - Indicating the Interrelating steps 
among component procedures and to orient subjects as to where 
they are at any point In their learning experience. 

c. A new, added FINAL EXERCISE 

The original FINAL EXERCISE volume contained eight separate 
final exercises. Each was devoted to a simple, separate devel- 
opment task. I.e., doing a task analysis, or writing objectives, 
or formulating an Instructional strategy, etc. These eight 
exercises (with the revisions cited earlier) are retained In 
the revision. A ninth exercise has been added. It requires 
subjects to select subject matter of their own and to go through 
al 1 the tasks In the development process including following 
all the front end analyses, the development of a program, and 
Its tryout and revision. This exercise Is designed to have 
subjects interrelate all development tasks and to practice them 
In a^-^rrect, total sequence. This kind of practice parallels 
the kind of performance they wM 1 ultimately be expected to 
engage In. 

Revisions In Instructions on Administrative Arrangements 

1 . Revisions In Instructions on the sequence of practice activities 

Because of the sheer size of the program and because of the require- 
ment of shifting from work with one volume (e.g., the HANDBOOK) to work 
with others (e.g., the WORKBOOK and the FINAL EXERCISE volume), subjects 
In the tryout were to some extent Initially confused about the correct 
sequence of exercise events. With experience the confusion disappeared. 
Nevertheless, changes were made in the User's Manual, the ORIENTATION 
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volume, and In the WORKBOOK Itself to make instructions about sequence 
clearer* Changes were made In the format with which the schedule was 
presented. The changes served more clearly to Identify which volume 
was to be referred to, which pages were to be read or worked on, and 
what activity was to follow next. 

2 . Revisions In organization of materials used in FINAL EXERCISES 

FINAL EXERCISES present subjects with pre-prepared materials and 
with materials the subjects have to prepare. The preparation of the 
latter materials depends on a review of the pre-prepared materials 
(representing completion of tasks which occur earlier In the development 
sequence). Subjects experienced some confusion about which materials were 
to be reviewed and which were to be worked on. The revisions of the FINAL 
EXERCISE volume was designed to resolve this confusion. Each exercise in 
the revision, consisted of two clearly Identified sub-sections. One was 
marked, '^materials to review*' and the other, "your assignment/* This 
distinctive identification was intended to provide better Initial 
guidance In the conduct of final exercises. 

3* Revisions in how to go through the program 

A review of error data and subjects' comments revealed a relatively 
small number of problems with program content and a moderate level of 
difficulty in knowing how to take the program. The complete revision of 
the ORIENTATION volume and the moderately detailed revision of the 
User's Manual (already cited) were made to deal with this problem. 
(Evidence - comments - from the tryout with Group III, which used the 
new ORIENTATION volume, Indicates that some of the difficulties 
experienced by earlier groups were diminished. This is also attributed 
to the reading of HANDBOOK sub-volume "A** before actually starting the 
program.) 



SUMMARY 



A considerable number of changes were made In the program. The 
changes were made In response to the types of difficulty which were observed 
(based on performance data and subject comments). 

The major changes were in Instructions and techniques designed to make 
it easier for subjects to know what to do in going through the program. 
Considerably fewer changes were made In materials designed to teach program 
content. This emphasis In revision reflects the priorities Identified In 
tryout observations. 
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Potential users of an Instructional program, when faced with a 
decision about the appropriateness of a particular program for their 
special purposes, have broad Information needs. Relevant to their decisions 
Is Information about the validity of content, the conditions under which 
the program was tried out, the results obtained, and, not least, the con- 
struction the developer himself puts on all this information. The next 
several sections attempt to provide this range of information about 'a 
technology for developing Instructional materials,' the title of the 
program which has undergone a developmental tryout which in turn is the 
subject of this report. 

Content Val idi ty 

The potential user of a program on Instructional design has every 
right to ask the question "why this design model rather than another?". 
For a performance so complex and large In scope as instructional design, 
the comparative, empirical study of alternative models becomes prohibitive. 
It would take a considerable amount of time and the cost would be 
prohibitive. In the absence of data on which to make a judgment about 
comparative merits of design iwdels, rational considerations provide a 
serviceable substitutee 

The present model Is based on a rationally performed, detailed task 
analysis of the design process (Gropper S Resnick, 197l)o The detailed 
procedures it prescribes, based on research and state-of-the-art consider- 
ations. Is unusually comprehensive. Tryout subjects who have studied other 
design models have judged this model to be considerably more structured, 
detailed, and comprehensive than the otherso Based on the available try- 
out data, it Is also possible to conclude that the model can be learned 
and performed with adequate proficiency. 

Does the model offer prescriptions which lead to effective instruction? 
The question is applicable to any design model and can be answered only by 
long term, programmatic research. As noted In the original, final report 
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(Grppper, 1971), It Is not an easily researched questlono The design 
process Is unusually long and complex. Student achievement Is the end 
product of a long series of Interrelated activities. It Is difficult to 
assess the contribution to effective Instruction of any particular design 
activity. The one exception concerns the contribution that can be made 
by the tryout and revision of instructional materials. Any model which 
Incorporates such a feature can expect, with varying number of tryout cycles, 
to bring about a desired proficiency level. Any model can be assessed 
for and differing models can, therefore, be compared for the levels of 
proficiency J:hey bring about 'pviov to revision and for the number of 
revision cycles they require. However, even such comparisons still leave 
open the question of how much the algorithms for each major task contribute 
to the overall effectiveness of a model. 

The data collected In the evaluation of the * Instruct lonal technology* 
program, concerns only the adequacy of the program to teach the use of 
the model. They provide no evidence to evaluate the adequacy of the model a 
As of this date, there -is on!y the rational analysis of the model to support 
a recommendation that it be used. The future, of course, requires empirical 
data to complement the rational evaluatlone 

Adequacy of the Training Vehicle 

it is not particularly meaningful to attempt to attribute an overall 
quantitative grade to the program. There Is no point In reporting the 
percentage of people attaining a particular level of proficiency. The 
performance Is too long and too complex for that kind of quantitative 
reporting. It Is more meaningful. It Is suggested, to look at parts of 
the program, to see what works and what doesn't, and to report on these 
differences. 

All told most of the program content worked well with relatively low 
error rates. This seems to be all the more remarkable in view of the 
typically large operant spans Involved. Participants often read any- 
where from five to thirty five or so handbook pages before turning to 
workbook exercises. Yet error rates on those exercises were acceptably 
low. For those portions of the program where error rates in excess of 



twenty percent were made (very few such portions were found), necessary 
changes were made. Thus, the developmental or formative data obtained 
were used to assess the adequacy and to diagnosethe nature of inadequacies 
of Individual sections of the program. That relatively few content changes 
were made (relative to the size of the program) suggests the adequacy of 
the program. 

The major changes that were made were largely administrative in nature. 
Because of the size of the program, the number of program components, 
the relatively complex sequence of events Involved In switching from 
component to component, novel formats, all contributed to student difficulties 
In worlcing with the program. While these difficulties diminished as students 
progressed beyond initial assignments, major changes were still intro- 
duced to provide students with a better orientation to the use of the 
program. Students In Group III had available a few of these changes 
(thus representing a partial cycle of tryouts). Lessened student difficulty 
was already noticed just on the basis of just a few of the revisions 
which were made (e.go, the new ORIENTATION volume and the scheduling of 
the reading the 'W Volume of the HANDBOOK before the first actual practice 
assignments). 

However, since only one formative tryout was conducted, the complete 
revised program has not been evaluated. There is no expectation that 
error rates on program content should be any worse. There is every 
expectation that it will be better. Based on partial results indicating 
somewhat better administrative instructions,, there is also every expecta- 
tion that use of the revised program will proceed more effectively and 
ef f i ciently. 

General IzabI 1 1 ty of the Findings 

Dependable statements about how this program is to be used and by 
whom must reflect the characteristics of the tryout sample and the con- 
ditions under which the program was administered. The following descriptive 
statements should be considered in deriving conclusions about the 
general izabll i ty of the findings. 
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1 • ret . The sample 

— participants were college students Involved In some form of 
advanced training 

— two of the three groups elected to take the program (as 
opposed to another option) when they elected to take a 
course that was being offered by the principal investigator, 
i.e. they were motivated to take the program; 

— the program was being taken as part of a course for which 
a grade was to be assigned (i.e. a motivational factor); 

~ participants had varying backgrounds, particularly varying 
degrees of experience in instructional technology; but, 
none had any experience with the design model involved In 
the current program; and 

~ three separate groups took the program, with all three 
producing roughly comparable results. 

2. re; The conditions of administration 

~ tryout participants worked at their own self-adopted pace; 
however, self-pacing occured within some time constraints; 
participants were required to complete weekly assignments 
and It was within the week-long period that they could 
adopt their own pace. , 

— A group discussion led by the principal Investigator 
followed each week-long work period. Participants 
received feedback and corrections. 

-- Group III took a partially revised program, including 
a revised sequence of activities. 

It appears reasonable to conclude that a college educated population,' 
with or without prior Instructional design training or experience can 
negotiate this program. The availability of a trained Instructional design 
to participate in periodic group discussions Is desirable. While the course 
was intended to be self-instructional, based on the tryout conditions 
which prevailed and on results (I.e. errors) noted prior to remediation 
during discussions, the conclusion to be drawn Is that, without further 
tryout in which no Instructor Is present, the program is not fully self- 
instructional. Some parts of the program do offer more of such a capability 
than others. 
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with an Instructor present, with a modified self-pacing capability 
possible, and with a college trained population this program can provide 
effective Instruction In Instructional design. 

Recommendat Ions 

The program on Instructional technology has undergone just one cycle 
of tryout and revision. The revised version clearly requires Its own 
evaluation and a substantially larger tryout group should participate 
In ito It should be tried out with other types of students and with other 
discussion leaders available. Having made these obligatory statements. It 
Is also fair to point out that, however small-scale this tryout of an 
instructional design program has been, this report on that program represents 
one of the few that Is data^based* While their are a number of available 
programs (and books) on the instructional design process, few have 
published data to support them. 

This report provides data to support the view that this program works 
moderately well. It is anticipated that the revised versions should do 
better. As such it is a transportable, self-Instructional (requiring 
some expert to provide periodic feedback) program capable of teaching 
a highly detailed set of Instructional design competencies. 
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Breakdown by Tasks 
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33% 






0% 


m 


100^ 


100? 


n=5 n=6 n= 


7 

J items 


>20% 








<20% 








0% 


100!^ 


100? 


100? 


n=5 n=6 n=3 


□ 

^3 items 


>20% 






15? 


<20% 




38% 




0% 


100? 


62% 


85? 


n=5 n=6 n=3 
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TASK 



SECTION # 


Error GROUPS 
Rate 1 II III 


LJ 

_! items 


>20% 






100% 


< 20% 








a% 


100% 


100% 




n=5 n=6 n=3 


□ 

items 


>20% 




06% 


18% 


< 20% 




70% 




0% 


100% 


2k% 


82% 


n=5 n=6 n=3 


□ 

items 


>20% 








< 20% 








0% 








GRAND TOTAL H- 11- H- 
"B" 


□ 

_zz_ items 


>20% 




06% 


19% 


< 20% 


06% 


36% 




0% 




58% 


81% . 



^ n=5 n=6 n=3 
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APPENDIX B 

Results of Final Exercises 
Groups I I and I I I 



B.I 



Resul ts for Group I I 
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FINAL EXERCISE #1: TASK j 

CRITERIA SUBJECTS 





#1 


#2 


#3 


#4 


#5 


#6 


ANALYSIS OF PROGRAM RESULTS 














Forms are Correctly Used 


Yes 


Yes 


Yes 


— 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


— 


Yes 


Yes 






























PROGRAM REVISIONS 














Content Entries are Valid 


Yes 


Yes 


Yes 




Yes 


Yes 
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FINAL EXERCISE #2: TASK I 



CRITERIA SUBJECTS 





#1 

JT 1 


#2 

IT 


#3 


#4 

JT ■ 


#5 


#6 


DEVELOPMENT OF PROGRAM 














Forms are Correctly Used 


Yes 


Yes 


Yes 


No 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


No 


Yes 





































































































ERIC 



FINAL EXERCISE '3: TASK ° 

CRITERIA SUBJECTS 





#1 

TT 1 


■n*. 


rr 0 


if A 


# u 


#u 


FORMULATING STRATEGIES 














Forms are Correctly Used 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 
















DEVELOPING A PROGRAM 














Forms are Correctly Used 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 
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FINAL EXERCISE TASK f 

CRITERIA SUBJECTS 





#1 


#2 


#3 


#4 


#5 


#6 


CONSTRUCTING A TEST 














Forms are Correctly Used 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 






























FORMULATING A STRATEGY 














Forms are Correctly Used 


Yes 


Yes 


Yes 




Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 




Yes 


Yes 
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FINAL EXERCISE #5: TASK h 



CRiTERiA SUBJECTS 





#1 


#2 


#3 


#4 


#5 


#6 


SIMULATION 














Forms are Correct Iv Used 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 






























CONSTRUCTING A TEST 














Forms are Correctly Used 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


f 
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FINAL EXERCISE #6: TASK d 



CRITERIA 



SUBJECTS 



#1 



#2 



#3 



#4 



#5 



#6 



STATING OBJECTIVES 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



No 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 
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Forms are Correctly Used 



Content Entries are Valid 



SIMULATION 



Forms are Correctly Used 



Content Entries are Valid 
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FINAL EXERCISE tT. TASK b 

CRITERIA SUBJECTS 





#1 


#2 


#3 


.#4 


#5 


#6 


ANALYSIS OF BEHAViOR 














Forms are used Properly 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


No 


Yes 


No 


Yes 


No 


Yes 




1 


























STATEMENT OF OBJECTIVES 














Forms are used Properly 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


No 


No 


Yes 


* 

Yes 
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FINAL EXERCISE #8: TASK b 



CRITERIA SUBJECTS 





#1 


#2 


#3 


#4 


#5 


#6 


ANALYSIS OF BEHAVIOR 














Forms are used Properly 


Yes 


Yes 


Yes 


Yes 


Yes 


Yus 


Content Entries are Valid 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 






























STATEMEMT OF OBJECTIVES 














Forms are used Properly 


Yes 


Yes 


Yes 


Yes 


Yes 




Content Entries are Valid . 


Yes 


Yes 


Yes 


Yes 


Yes 
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Results for Group 1 1 1 



/ 
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FINAL EXERCISE #1: TASK j 

CRITERIA SUBJECTS 





#1 


#2 


#3 


ANALYSIS Or rKUlSRAn KcSULIS 








Forms are Correctly Used 


Voc 


T eb 




Content Entries are Valid 
























PROGRAM REVISIONS 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Y«i 


Yes 



























FINAL EXERCISE n-. TASK I 

CRITERIA SUBJECTS 





#1 


#2 


#3 


DEVELOPMENT OF PROGRAM 








Forms are Correctly Used 




Yes 


Yes 


Content Entries are Valid 




Yes 


No 
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Fir^AL EXERCISE n-. TASK g 



CRITERIA 



SUBJECTS 





#1 

ffl 


ffZ 


#3 


FORMULATING STRATEGIES 








Forms are Correctly Used 





Yes 


Yes 


Content Entries are Valid 





Yes 


Yes 










DEVELOPING A PROGRAM 








Forms are Correctly Used 


N.A. 


N.A. 


N.A. 


Content Entries are Vallc 






Yes 
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FINAL EXERCISE TASK f 

CRITERIA SUBJECTS 





ifl 


if 9 




CONSTRUCTING A TEST 








Forms are Correctly Used 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


















FORMULATING A STRATEGY 








Forms are Correctly Used 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 
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FINAL EXERCISE #5: TASK e 



CRITERIA S!!BJECTS 





#1 


#2 


#3 


SIMULATION 








Forms are Correctly Used 


Yes 


Yes 


Yes 


Content Entries are VaHd 


Yes 


Yes 


Yes 


















CONSTRUCTING A TEST 








Forms are Correctly Used 


Yes 


Yes 


^s 


Content Entries are Valid 


Yes 


Yes 


Yes 
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FINAL EXERCISE #6: TASK d 

CRITERIA SUBJECTS 





#1 


#2 


#3 


STATING OBJECT- VES 








Forms are Correctly Used 


Yes 


Yes 


Yes 


Content Entries are Valid 


No 


Yes 


Yes 




• 














SIMULATION 








Forms are Correctly Used 






Yes 


Content Entries are Valid 






Yes 
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FINAL EXERCISE #7: TASK b 

'is 

CRITERIA SUBJECTS 





#1 


#2 


#3 


ANALYSIS. OF BEHAVIOR 








Forms are used Properly 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


Yes 


















STATEMENT OF OBJECTIVES , 








Forms are used Properly 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


No 
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FINAL EXERCISE #8: TASK b 

CRITERIA SUBJECTS 





#1 


#2 


#3 


ANALYSIS OF BEHAVIOR 








Forms are used Properly 


Yes 


Yes 


Yes 


Content Entries are Valid 


Yes 


Yes 


No 




- 














STATEMENT OF OBJECTIVES 








Forms are used Properly 


Ves 


Yes 


Yes 


Content Entries are Valid 


Yes 


No 


Yes 
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APPENDIX C 

Tryout Subjects- Comments; 
OVERALL EVALUATION OF: 
(1) MODEL 

(2) VEHICLE DESIGNED TO TRAIN OTHERS TO IMPLEMENT 



Subject #1 



The program for developing Instructional materials presented In this 
course could be very usefuK I am not convinced that the backward chain- 
ing approach Is the best way to present the material. The backward chain- 
ing In the first three or four handbooks was confusing in refering to 
concepts and forms not yet encountered. The instructions in these handbooks 
to read long sections of other handbooks was also awkward. It was only 
after Handbook F that I was aware of any benefits of backward chainingo I 
understand that this may be due to familiarity with the entire program 
format (diagramming) and. not just backward chaining. 

The use of diagrams may be more useful with some students than others. 
I had a tendency to read the diagram and then try to put It into prose. I 
do believe it should be emphasized that students do not have to learn 
everything in a diagram before moving to the next. 

It was not clear how much of the information presented in the program 
was based on empirical research and how much represented hunches of the 
author. Per'haps.a handbook of references could be provided. 

The examples were generally good. They were varied and for the most 
part clear. The few exceptions were some of the early examples on economic 
pol Icy. 

The final Exerciset, were helpful in understanding the application of 
the material presented in the handbooks. I feel I would have benefited by 
some kind of exercises after the last final exercise which carried one 
problem through all the steps. 

I would use the program as a Job aid, especially Handbooks G, B, and C. 
I was disappointed in Handbook H. Does the author^feel that individual 
differences are just not that important or that there is not enough known 
to say anything? What about treatment-interaction considerations? 

I doubt Just anyone could pick up the program and use It. The 
terminology is for persons in areas other than social sciences would be 
difficult. 

Overall I found the program useful and of potential value as a Job aldo 



Subject #2 
General Evaluation of Prociram 



In evaluating the program, the major criteria used was the relation- 
ship between the planning stages and the development stages. That Is, are 
the different types of analyses - task, learning, mod<a, and different 
clcsses of tasks, learning, modes, used and dealt with differently in the 
design and development stages. It appears that in relation to this 
criteria the program Is very good. All analysis (task, learning, modes) and 
classifications of types of tasks, etc, are consistently used throughout 
the program In designing and development of Instructional program^ 
(handbooks B and 6), 

If at some future time, t am employed to design and develop an Instruc- 
tional program, I would probably use Handbooks A, B, C, to begin with and 
then use sonne of the main concepts presented In Handbooks G and I, I 
especially like tne ideas presented concerning the sequencing of tasKS and 
many of the instructional strategies. 

Subject #3 
Over^al 1 Evaluation 

The program is very comprehensive leaving few stones unturned. 

As mentioned In the "Orientation" the optimum learning situation for 
the program is weekly meetings for critique and discussion. As a matter 
of fact, I suspect that If one was to attempt the program individually 
without the group interaction and counseling, he would give up before 
Handbook G, At that point. Insufficient frjmiliarity has been gained to 
over-ride roughly 18 inches of material left to go. The outlook at that 
point is very demoralizing. It should be added quickly that in retrospect 
It Is not nearly that bad. For a!i the length of the program, J don't 
believe it can be reduced significantly. 

If the program is to be implemented In in-service education for 
teachers, it should be broken into some smaller pieces or spread over a 
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Subject #3 (continued) 



longer time span ( a minimum of 2 quarters and maybo 3) to allow the full- 
time teacher to go through the program. 

The knowledge of the technology Is much more Important for the public 
school teacher than the competency. I seriously doubt that the tendency 
could be Implemented more than piece-meal in the public school. This 
situation is probably reversed in an industrial or military training 
situation where the teacher is more atuned to the competency he is teach- 
ing than the competency of teaching. 

For me, the program has more value since I intend to return to the 
military as an instructional technologist. However, should I return to the 
public schools, its value will decrease. 

Part(s) 1 would use ; 

J. Evaluate instructional Materials 
G. Formulate Instructional Strategies 

Subject ^ 
No Return ' : 

Subject #5 
A Discussion of the Cropper Mode! 

The model Is somewhat voluminous^ In the amount of material presented 
and thus It would possibly take several learning situations (where the 
model Is completely followed through) before one could become comfortable 
with It in total I ty. 

Although the Index Volume Is a repeat of the handbook's contents \ 
st\\\ feel that some overall flow diagram or flow chart (s) should be 
available to the uss^;" to enable him to look at the overall picture and then 
focus In on the particular area (handbook) that he or she might feel they 
wanted to use. 



Subject #5 (continued) 

At a less global level certainly many of the handbooks are quite well 
done and quite Informatl ve~example B & G but others are somewhat trivial 
and as has been satd In class are so common sens leal that they should be 
eliminated or at least reduced or combined* 

Since the orientation manual Is being rewritten perhaps this will 
straighten out the confusing problem in the beginning of the learning 
situation where we, as students, did not know what we were to accomplish* 

Backward Chaining 

As I stated In the other paper I wrote on what I would use from the 
model 9 I found the use of backward chaining quite useful* <t enables the 
learner (not necessarily the job holder) to see the results of what should 
be done and thus work In reverse order to put these component parts together* 
As I suggested In class, If you had a sufficient. size of'student population^ 
you could empirically test which approach might do the job better—forward 
or backward chaining* 

In conclusion, I certainly hope to use portions of the nw^del in any 
Instructional design I may be involved In* Also thank you for the material 
as I am more firmly convinced this approach in the class was more valuable 
than the alternative selection you gave, even though you lost some students 
because of it* 

Subject #6 

Some of the more useful content or recommendations included: 

1* The program's consistency and repetition of concepts. Its organization 
and its effective presentation* As a result, I have come out of the course 
with a better: 

a* orientation to the Instructional development process^ 
b* way of classifying educational problems 
c* grasp of the vocabulary and concepts In ed* research 
2* The use of diagramming as an effective and efficient device* 
3» Al I of Handbook G* This handbook will always be the most meaningful 
and helpful section* Probably as complete as I could want for many years* 



Subject #6 (continued) 



4« Handbook E. A solld^ complete look at simulation. Again, will refer to 
that section whenever any question about simulation arlseso 
5. The breakdown of learning into the four basic categories*, The break- 
down was amenable to concept acquisition research, which Is a current 
interest of mine. 

In summary, Handbook E S G were the most solid, effective and efficient 
handbooks. 

Handbooks A, B, D, F, I, J made important contributions. However, 
I believe that A & B should be combined. 

Handbooks C S H were least effective. I disagree with the arbitrariness 
of C (rationales can be arbitrary) and with the Inconsistency between H and 
the model (as Implied In the other handbooks). 

As to your model, I believe I would use much of your materials If I 
was teaching an instructional development course. I would throw out A, 
shorten B, throw out most of C, shorten D, keep E> F, S G as is, disburse 
H throw out, shorten I & J. I would also Include a model description 

(and present It first), add a section on documentation and possibly, a 
section on the relationship of research and development (how the two can 
be integrated and how they usually coexist). I would definitely make It 
branched, leaving out whole sections for some Individuals If students showed 
competency on some kind of pretest* Herb, Jerry, Pam and I have had 
courses dealing with objectives—which also included the Idea of specifying 
the type of learning necessary within the objective. 

The backward chaining In the model and the diagramming throughout was 
excellently thought out and carried through. I thought the Final Exercises 
were challenging and they supplemented and complemented the backward chaln- 
Ingo I learned much from your examples, the thoroughness and the con- 
creteness helped there. Back to backward chaining, I believe the same effect 
could not have been accomplished if the handbooks were forward chained. 
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APPENDIX D 
Tryout Subjects' Comments: 
COMPARISON WITH OTHER MODELS 



D.l 



Subject #5 
Model Comparisons 



In comparing Dr. Brigg's model with Dr. Cropper's model some of the 
main differences are as follows: ' 

1. Brigg's model (at least the manner we approached it in class) used 
feedback only from him analyzing It in great detafl. The results are un- 
testable until the end when summative and evaluative testing can be done 

as in Cropper's model. Cropper's model approach Is the backward chain and 
thus able to get analysis of output quickly plus the class verbalization 
is good. It would be better If we received written feedback from Cropper 
on hfs analysis of our outputs. 

2. The forms used In Brigg's model were to minimal. 

3« The final example (products) In Brigg's model were very good. 

In comparing Dr. Dick's model (used In EDR 537) the approach was very 
different^ 

1. Dr. Dick's model used adjunct program Instruction (P1) which 
assigned readings then quizzed you on the information via CAI terminals. 
At the completion of all materials then one used the information to develop 
and test a PI In some material of our choosing. The advantage of this 
approach over Briggs and Cropper was the use of many other individuals 
thoughts on the area being studied. The disadvantage was that It was far 
less structured than Briggs and would be considered completely unstructured 
In comparison to the material of Cropper's model. 

What I Would Use in a Design Course 

From Dr. Cropper's course I would make use of the forms on task 
description, specifications of objectives, test development, plan for 
instructional strategies, developing Instructional materials, and summary 
of program errors. J would also use the backward chain approach as In many 
cases seeing the total picture first then working on parts of it are more 
helpful than by starting at lower levels. 
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Subject #6 
Model Comparison 

EDR 537: Techniques of Programmed Instruction 

The ''systems approach'' embodied In this course was based on the 
Dick model (see Fig. lA), The course was organized into two parts. The 
first part looked at the component steps in the model in detail. Each step 
in the model was introduced by a set of readings (Mager, DeCecco, Gagne, 
Markle, etc.). Thus, the first half of the course was quite theoretic. 
The second half of the course was devoted to actually developing a PI text. 
The text was documented (steps and rat lonale fu I ly described), including 
a formative and summative evaluation. 

EDR 539: Adaptive Instruction Models 

The content here was at a more "macro-level." No practical instruc- 
tional development experience was needed or taught. The goal of this 
course was to take and develop~an "Adaptive Instructional System." The 
emphasis was on how the objective (goals), the personological characteristics 
of learners, the instructional packages and the systems needs (time, cost, 
and resource constraints) interacted, i.e., could be handled simultaneously. 

What Will Remain 

As a preface, let me explain a few matters. First, I am not interested 
In becoming an instructional developer. However, I will need to evaluate 
and use instructional developers* output, as I aspire to become a researcher/ 
coordinator In a public school system and/or at the University level. 
Second, I have a strong bias against the model used In the course (and I 
guess, that part of a person who advocates such a model) because it does 
not allow adaption to individual d i f f erences . . .not even in terms of repetition 
and review. Handbook H Is weak, and the ideas there expressed are not 
foP ^ed up in the other handbooks. In fact, the model and the content would 
be mw - consistent if Handbook H were deleted. 



D.3 



While many specific recommendations were useful, I will use and 
remember the Dick model because It Is more amenable to my view and goals. 
I believe that sequence, step size, Instructional strategy, and media type 
can be adapted (varied) to match Individual learners so to facilitate 
their learning, I choose the Dick model because I am most of al I a researcher, 
and see the development of Instruction as a step resulting from research. 
The seemingly arbitrary decisions concerning error rate, step size, media 
type, amount of practice and review were, for the most part, rejected. 
There are more research questions, as far as I am concerned. 



APPENDIX E 
Tryout Subjects* Comments On: 

INDIVIDUAL HANDBOOK SUBVOLUMES 



E.I 



COMMENTS ON HANDBOOK SUB-VOLUME "A" 

Subject #1 
No Return, 

Subject #2 
Handbook A 

I found A to be very thorough. If I was "on the job," i,e,, had to 
design an instructional program, I would turn to this handbook first and 
probably follow the general steps. 

Don't fully understand why you had domains: knowledge, performance, 
cognitive, personal/social behavior, and then not attack them (pg, 41), 

Subject #3 
Handbook A 

Handy - but should have been noted at each opporet, in earlier volumes 
(referring to A,5)» 

Subject M 
Handbook A 
New, useful Ideas on conceptSo 

a. criteria for identifying four types of criterion behavior, 
(pp. 6-7). 

b. Labeling systems for cross-referencing purposes (75). 

c. The whole step A, 5. 2(a) and A, 5. 2(b), 

It would be important to read some sections of A Handbook before 
starting the other Handbooks, e,g,, Step A. 5.2 (pp, 77,78,79,81 ,83, 9^...) . 

HANDBOOK A; PLAN STUDY OF CRITERION BEHAVIORS 

—Criteria for Identifying four types of criterion behavior: knowledge 
comain, performance, general cognitive skills, personal/social 
behavior (6-7) 



--Determining how to obtain Information necessary to describe or 

generate a model (25) 
—Information-gathering techniques likely to be used for different 

types of criterion behavior (27) 
—Selecting sources of Information appropriate to types of 

criterion behavior and types of Information-gathering techniques (35) 
—Five deslderable properties of Informatlon-col lection techniques 

Examples (60) 

—Recommended labeling system for cross-referencing purposes (75) 
—Sub-Step A. 5.2 (a): Selection of forms used In: 
• a. Task description 

b. Steps 

c. Sub-steps 

d. Task Analysis 

Yellow forms included (pp. 66-102) 
—The whole step A. 5 gives a very useful Information for Information- 
collecting Instruments and procedures (pp. 53-166) 

Before reading Handbook B. 1 would suggest to read Workbook, p* 115, 
Ex. 2A for a good summary and examples of Discriminations, Generalizations 
and Associations. (See also pp. 1-16 and 1-17). 

Subject #5 
A« Plan study of criterion behavior 

1. Fairly easy to read 

2. The examples of labeling are rather confusing at first glance 
and perhaps the whole topic of cross-referencing Is so basic 
that people should know outline techniques, without showing It 
here. 

Subject #6 

good volume; but valuable to me much more as job aid when job Is 
being actually undertaken 

had little patience to stick with the concepts and the verbage. 



COMMENTS ON HANDBOOK SUB-VOLUME "B" 



Subjects /yi and n 
No Return 

Subject n 

This volume was straight-forward, I can't remember anything out of 
the ordinary. Again, I feel that the back-chaining is having an effect to 
ease the Impact of any new material. 

Subject §k 

HANDBOOK B ; Collect and analyze data about criterion behavior 

--Different levels of specificity in describing behavior (28) 
Criteria for determining the adequacy of the size of the sample 

of incidents (30) 
—Contingent and non-contingent behav'ors (35) 

—Criteria for Identifying contingent and non-contingent behavior 

(task, step or sub-step) (p,57) 
—See also the whole sub-step B,2,2,: 

+ Determining the order of Information collection about "performance" 

that is contingent (Illustration) (pp, 58-59) 

+ Determining the order of information collection about "performance" 
that is non-contingent (pp, 60-61) 

+ Determining the order of information collection about "knowledge 
domains" (pp, 62-63) 
—Determining the size-of functional units to treat as and label as 
"TASKS" (79) 

—Criteria for Identifying functional units at differing levels of 
general i ty (8A) 

—Illustration summarizing how to sub-divide knowledge domains (lOA) 

--Inputs and Actions related to description of terminal behaviors (107) 

— Specific Inputs vs. Input class 

Specific " actions vs, action cTass (p#l08) 



Subject ffk (continued) 

—Criteria for Identifying when exhibition of terminal behavior after 
instruction of training requires RECALL vs^. TRANSFER (110) 

--See the whole sub-step B.A.2.: Identify and diagram discriminations, 
generalizations and- associations^ Key point in these materials. 

—The whole step B.^.^.: Critferia for Identifying difficulty In 
discriminate, associate, generalize, chaining. .. (I968-I85) 

Subject ^5 

1. Very Informative. 

2. Fairly easy to read and comprehend 

3. The discussion of the flew of Information from forms A. 5 are very 
well done. 

4. The examples in the A. 2 sections on Tasl< Analysts are quite good. 

5. Although I thought I understood B.5 on Mode Analysis, my workbook 
answers were not usually right and I feel there could be some 
room for judgment responses in type of mode presented. 

Subject #6 

1. This volume operat lonal i zes what was A. \ believe that Handbook 
A can be deleted-— or altered a bit and used as an Introductory, orienting 
handbook. B is strong enough to stand alone. 

2. Well organized volume— -it Is a bit difficult to evaluate because 
I had gone through many previous task analyses and It was a bore to get 
nothing that much different or new--except a numberlng-categorizing scheme. 

3. The forms are excellent for the task analysis. The forms should 
be Introduced a bit earl ier— probably when the types of learning are 
introduced. They are (the forms) helpful in seeing the '*whole.*' 



COMMENTS ON HANDBOOK SUB-VOLUME 'V 



Subject ^1 
No Return 

Subject #2 
Handbook C 

I found *'C" to be straight forward and easy to understand. I partic- 
ularly liked the analysis of the three types of possible re 1 atikon ships 
among sub-criterion behavior (pp.28) and among crrterion beh'ffitrior iCqap* 48)v, 
and relationships between 3 types. 

Subject ff3 
Nothing major comes to mind. 

Subject A 

New interesting concepts: 

a. Vertical and horizontal task analysis (po20) 

b. mentions sub-cr 5 terion behaviors cm) bear to one another (32—33) 

I found a contradiction: 

p. 3^, e«g*> bottom 2nd column 

see also p. 52, #2. 2nd column 

Additional Comments: Subject M 

HANDBOOK C ; Sequence and Group Criterion Behaviors 

—Determining how to make sequencing decisions about vertical and 
horizontal task analysis results (20) 

--Conditions and Examples illustrating three relationships sub- 
criterion behaviors can bear to one another (prerequisites, provid- 
ing inputs to one another, sharing common elements) pp. 32-22. 
See other examples of this relation (p. 52) « 
In general, the whole step C.I. 2 (pp. 27"^0) is very useful. 

--Determining how to sequence these different relations (3**) 



Subject #5 

I did these books 12 week ago and thus am trying to recall problems 
from several weeks ago. 

SEQUENCE AND GROUP CRITERION 

1. Very verbal and perhaps too much so. 

Subject #6 

Sequence appears obvious enough a topic (to one who has considered 
It) to warrant a briefer handbook 

More concrete examples would help more than the **ideas*' and "General 
suggestions" put: forth j 



COMMENTS ON HANDBOOK SUB-VOLUME "D** 



Subject ^1 

There were some pages (examples) that were hard to read because of 
small print - also in workbook. Found confusing terms self-contained and 
not contained, modified and unmodified preparatory objectives, examples of 
objectives given - information to be included in objectives good. 

Subject #2 

Appear to be good analysis for writing objectives. Is consistent with 
other portions of the program. 

Subject #3 

I still don't know the difference between "modified" and "intact" 
preparatory ob^Jectives 

Didn't find anything on "You Must Learn to Box" on form D.2(2) 
Otherwise very straight forward 

Subject M 

— pg. 535 Determining how to write objectives. I missed minimum 

criterion performance here. 
—Are there any difference between a) criterion and self-contained? 

b) preparatory and not self-contained? 
"Interesting: concept of forward, reverse and both directions. 

objectives (p. 32) 

Additional Comments Subject Ifk 

HANDBOOK D ; State Criterion and Preparatory Objectives 

—Criteria for identifying Criterion Objectives and Preparatory 
Objectives (6) 

--Examples illustrating situations in which objectives will require 

forward, reverse or both directions (32) 
—Determining whether to plan for (and include in a statement of 

objectives) two performance directions (33) 

ERIC . , I 



Subject #^1 (continued) 

--Determining how to write objectives (53) 

—Criteria for Identifying self-contained vs. not self contained 

preparatory objectives (66) 
— Include in objectives (92) 

a) Discriminations ; 

--among Inputs 
--among outputs 

b) General izations ; 

—Across Inputs 
—Across outputs 
--Across actions 

c) Associ atons/cha i ns ; 

--between inputs 
-- between actions 

Subject ffS 

lo The new term preparatory objective Is not a standard term and thus 
the transfer of prior terminology to this can tend to confuse although not 
too much* 

2. Examples are very good. 

3. Part D,2 - Preparation of statement of objectives is very well done. 
h. Easily read handbook. 

Subject H 

Overall: not as helpful as most handbooks 

Except for the "You Must Learn To" in the objectives for students, is 
common sense for the most part. Having learned objectives last year, 1 
would have rather worked the task analysis first. ♦.or both together. 
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COMMENTS ON HANDBOOK SUB-VOLUME "E'* 



Subject #1 

Didn't find that much Info In E or final exercise* What was said 
needed to be lncluded--beyond that--perhaps this info# could be included 
as part of another handbook* 

Subject n 

! didn't get much of anything out of Handbook E, It seems that most 
all the information and decisions are obvious. 

Subject n 

''E'' is thorough and adequate - simulation problems do not arise within 
the context of '"E". They arise from a knowledge of the subject* There- 
fore requiring the almost constant association with an "expert." 

Subject M 

E Volume: 

A) the most useful material: 

a) summary of procedures on pages 19,3^, and 48 

b) examples of simulation varying in degrees of logistical burden due 
either to cost or to the administration considerations (p«57) 

c) I think all the volumes read before were much more interesting 
(G or I for example). 

d) I would include this content in Volume F. 

Additional Comments Subject itk 

HANDBOOK E t Plan Simulation based on instructional and logistical needs. 

—Summary of procedures for determining and recording the serious- 
ness' of the need to simulate (18) 
--Summary of procedures for identifying and recording criterion 
properties (3^) 

—Summary of procedures for planning alternative forms of simulation 

to meet instructional goals (48) 
— Examples of simulation varying in degrees of logistical burden due 

ERXC either to cost or to administration considerations (57) 



Subject ff5 

A fairly easy handbook to read. Examples were good. Final Exercise 
not very challenging. 



Subject H 

Good progression; clear; Right length/depth. 
—Problem: 

When considering logistical and instructional needs, should 
make a statement that points out that it Is important to estimate 
how much it would cost If a student did not learn. 
"Categories are somewhat arbitrary; Sample vs. Standardized difficult 
to distinguish, at timeso 

Deciding about manipulation can be the same as deciding about 
simulation. 
--Danger/damage (and other categories) 

One must also consider the effect of not learning (on danger, 
damage, cost). 
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COMMENTS ON HANDBOOK SUB-VOLUME "F" 



Additional Comments Subject ffk 

HANDBOOK F t Develop Diagnostic and Evaluative Tests 

"Product and Process and "criteria for identifying three types 

of outputs (27-28) 
—Criteria for identifying how a test Item samples criterion behavior 

(kk). See also the whole sub-step F.2.3. (pp* ^3-59) 
"Illustration of Procedures Involved in preparing test Items (58) 
—Determining how to Interpret general errors patterns regarding 

discriminations, generalizations and associations (84-85) 
—Determining the types of probes to use to determine source of errors (IO8-I09) 
--Summarizing procedures Involved in deciding whether to develop 

diagnostic probing procedures (116) 
—Criteria for identifying what it is you want to find out by 

using diagnostic test items. 
— Determining how to develop recogni tl^in' test items to diagnose 

specific types of learning problems (12A) 
— Summarizing procedures involved in preparing diagnostic test items (128) 
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COMMENTS ON HANDBOOK SUB-VOLUME "G" 



Additional Comments Subject M 

HANDBOOK G: Formulate Instructional Strategies 

--Types of predominant learning and performance problems (p. 41) 
—Examples illustrating this (po44) 

— Criteria for identifying five types of preparatory practice 
progressions: 

+Unit Size 
+Hode 

+Prompt ing/fad ing 
^Content 

+Frequency variation (pp. 7^-75: these two pages are the key for 
understanding the hole sub-step (G,2.1.) 

—Determining where to obtain information needed for designing 

preparatory practice progressions (pp.153 and ff) 

--The job of progressions ; 

+Job Diagram overall view (p. 70) 

+What is the purpose of progressions (73) 

+How to fill the forms (116,119,120) 
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APPENDIX F 
Tryout Subjects* Comments On: 
DIAGRAMMING 



F.l 



Subject if\ 



I find it much easier. to use the diagrams now than where I began the 
materials. There is still the tendency to read the diagrams to *Mearn'' 
everything there« Find most useful the Job Diagram page In organizing 
materIal--Se)dom use the diagram on Description of Substep diagram. Forms 
referred to by a number i.e., "use form A. 5 for infoo In filling out form" 
are confusing. 

Subject ^2 

At first I tried to retain too much of the information presented or 
the sequence of the tasks. Spent tremenjdous amounts of time. Now I am 
only revising the steps and putting Into my own words what I am supposed to do 
(Job diagram, etc.). On diagram presenting information about a component 
(e.g., type of learning or procedure) I am now just sicimming over at first 
and spending more time on them while doing work book exercises and 
especially on the final exercises. I do not seem to get much information 
out of diagrams defining or describing something, t find the amount of 
different kinds of things being covered at one time to be distracting. 
I seem to miss alot of Informationo However, when 1 do the final 
exercises, I try to follow the step by step procedure presented in the 
handbook. It Is at this time that t seem to learn the most and. ''fit" 
things together. 

Subject ^3 

I like this type of presentation of material* I'm not textbook- 
oriented and therefore appreciate the organization which allows me to 
delete as I forsee the opportunity. One point - I don't see the need of 
pps. such, as F.26 since they serve mainly for reference purposes and 
appear in the index volume. 
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F.2 



Subject itk 



I have found the diagrams useful and instructional fn generaU Some 
of them are much more vaiuable and plenty of material, e.g., Handbook F, 
ppo84-85; pp. 108-109. The mere fact that the content is presented In a 
non-tradi t fonal way makes the subject more Interesting. However, some 
of the diagrams are trivia! and repeat Information already given in prior 
and some Individual diagrams. May be this will be useful in the future as 
a job aido 

Subject §S 

The Diagrams in general are somewhat complex 

Initially most diagrams were quite difficult to use 

Now reading is easier In most cases 

Content layouts - Good 
Step ** - Good 

Page Index - Poor and Confusing 

Substep Layouts - Fair 
Job Dlagr^n - Good 
ID Matrix - Fair 

Decision Matrix - Fair 
Complet ion Good 
Standards Matrix- Fair 
Examples & Ml us. Good 

Subject #6 
Diagramming: 

General: Useful, directs attention, emphasizes the whole and the parts, 
forces author to be complete, most importantly, puts knowledge into qualitative 
and for quantitative heirarchies which make diagrams more useful to learners 
of al 1 levels. That adaptive quality not found in prose (that is why use 
high-.l I ter) . Especially useful for those people who do not think in terms 
of words, but pictures. 

Specific: 

Standard Matrix: not as useful as it could be; present earlier and 
present when presenting forms, especially, useful for concept attainments 



